{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# arxiv-txt.org for literature reviews\n",
    "\n",
    "In this example, we will use [arxiv-txt.org](https://arxiv-txt.org) to generate a literature review for a given topic.\n",
    "First, let's identify a list of relevant papers on a topic we want to summarize.\n",
    "Here we will focus on \"Masked Autoencoders\".\n",
    "\n",
    "Here is a list of papers we will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "paper_list = [\n",
    "    \"https://arxiv.org/abs/2205.09113\",\n",
    "    \"https://arxiv.org/abs/2304.00571\",\n",
    "    \"https://arxiv.org/abs/2211.09120\",\n",
    "    \"https://arxiv.org/abs/2212.05922\",\n",
    "    \"https://arxiv.org/abs/2301.06018\",\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define some helper functions to get the summaries of the papers for the lit review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting summary for https://arxiv.org/abs/2205.09113:\n",
      "Getting summary for https://arxiv.org/abs/2304.00571:\n",
      "Getting summary for https://arxiv.org/abs/2211.09120:\n",
      "Getting summary for https://arxiv.org/abs/2212.05922:\n",
      "Getting summary for https://arxiv.org/abs/2301.06018:\n",
      "Summaries:\n",
      "---\n",
      "\n",
      "# Masked Autoencoders As Spatiotemporal Learners\n",
      "\n",
      "## Authors\n",
      "Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, Kaiming He\n",
      "\n",
      "## Categories\n",
      "cs.CV, cs.LG\n",
      "\n",
      "## Publication Details\n",
      "- Published: May 18, 2022\n",
      "- arXiv ID: 2205.09113v2\n",
      "\n",
      "\n",
      "\n",
      "## Abstract\n",
      "This paper studies a conceptually simple extension of Masked Autoencoders\n",
      "(MAE) to spatiotemporal representation learning from videos. We randomly mask\n",
      "out spacetime patches in videos and learn an autoencoder to reconstruct them in\n",
      "pixels. Interestingly, we show that our MAE method can learn strong\n",
      "representations with almost no inductive bias on spacetime (only except for\n",
      "patch and positional embeddings), and spacetime-agnostic random masking\n",
      "performs the best. We observe that the optimal masking ratio is as high as 90%\n",
      "(vs. 75% on images), supporting the hypothesis that this ratio is related to\n",
      "information redundancy of the data. A high masking ratio leads to a large\n",
      "speedup, e.g., > 4x in wall-clock time or even more. We report competitive\n",
      "results on several challenging video datasets using vanilla Vision\n",
      "Transformers. We observe that MAE can outperform supervised pre-training by\n",
      "large margins. We further report encouraging results of training on real-world,\n",
      "uncurated Instagram data. Our study suggests that the general framework of\n",
      "masked autoencoding (BERT, MAE, etc.) can be a unified methodology for\n",
      "representation learning with minimal domain knowledge.\n",
      "\n",
      "---\n",
      "# DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking\n",
      "  Tasks\n",
      "\n",
      "## Authors\n",
      "Qiangqiang Wu, Tianyu Yang, Ziquan Liu, Baoyuan Wu, Ying Shan, Antoni B. Chan\n",
      "\n",
      "## Categories\n",
      "cs.CV\n",
      "\n",
      "## Publication Details\n",
      "- Published: April 2, 2023\n",
      "- arXiv ID: 2304.00571v2\n",
      "\n",
      "\n",
      "\n",
      "## Abstract\n",
      "In this paper, we study masked autoencoder (MAE) pretraining on videos for\n",
      "matching-based downstream tasks, including visual object tracking (VOT) and\n",
      "video object segmentation (VOS). A simple extension of MAE is to randomly mask\n",
      "out frame patches in videos and reconstruct the frame pixels. However, we find\n",
      "that this simple baseline heavily relies on spatial cues while ignoring\n",
      "temporal relations for frame reconstruction, thus leading to sub-optimal\n",
      "temporal matching representations for VOT and VOS. To alleviate this problem,\n",
      "we propose DropMAE, which adaptively performs spatial-attention dropout in the\n",
      "frame reconstruction to facilitate temporal correspondence learning in videos.\n",
      "We show that our DropMAE is a strong and efficient temporal matching learner,\n",
      "which achieves better finetuning results on matching-based tasks than the\n",
      "ImageNetbased MAE with 2X faster pre-training speed. Moreover, we also find\n",
      "that motion diversity in pre-training videos is more important than scene\n",
      "diversity for improving the performance on VOT and VOS. Our pre-trained DropMAE\n",
      "model can be directly loaded in existing ViT-based trackers for fine-tuning\n",
      "without further modifications. Notably, DropMAE sets new state-of-the-art\n",
      "performance on 8 out of 9 highly competitive video tracking and segmentation\n",
      "datasets. Our code and pre-trained models are available at\n",
      "https://github.com/jimmy-dq/DropMAE.git.\n",
      "\n",
      "---\n",
      "# AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with\n",
      "  Masked Autoencoders\n",
      "\n",
      "## Authors\n",
      "Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami, Mehdi Nikkhah, Motilal Agrawal, Vishal M. Patel\n",
      "\n",
      "## Categories\n",
      "cs.CV, cs.AI\n",
      "\n",
      "## Publication Details\n",
      "- Published: November 16, 2022\n",
      "- arXiv ID: 2211.09120v1\n",
      "\n",
      "\n",
      "\n",
      "## Abstract\n",
      "Masked Autoencoders (MAEs) learn generalizable representations for image,\n",
      "text, audio, video, etc., by reconstructing masked input data from tokens of\n",
      "the visible data. Current MAE approaches for videos rely on random patch, tube,\n",
      "or frame-based masking strategies to select these tokens. This paper proposes\n",
      "AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our\n",
      "adaptive masking strategy samples visible tokens based on the semantic context\n",
      "using an auxiliary sampling network. This network estimates a categorical\n",
      "distribution over spacetime-patch tokens. The tokens that increase the expected\n",
      "reconstruction error are rewarded and selected as visible tokens, motivated by\n",
      "the policy gradient algorithm in reinforcement learning. We show that AdaMAE\n",
      "samples more tokens from the high spatiotemporal information regions, thereby\n",
      "allowing us to mask 95% of tokens, resulting in lower memory requirements and\n",
      "faster pre-training. We conduct ablation studies on the Something-Something v2\n",
      "(SSv2) dataset to demonstrate the efficacy of our adaptive sampling approach\n",
      "and report state-of-the-art results of 70.0% and 81.7% in top-1 accuracy on\n",
      "SSv2 and Kinetics-400 action classification datasets with a ViT-Base backbone\n",
      "and 800 pre-training epochs.\n",
      "\n",
      "---\n",
      "# Audiovisual Masked Autoencoders\n",
      "\n",
      "## Authors\n",
      "Mariana-Iuliana Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, Mario Lucic, Cordelia Schmid, Anurag Arnab\n",
      "\n",
      "## Categories\n",
      "cs.CV, cs.SD\n",
      "\n",
      "## Publication Details\n",
      "- Published: December 9, 2022\n",
      "- arXiv ID: 2212.05922v3\n",
      "\n",
      "\n",
      "\n",
      "## Abstract\n",
      "Can we leverage the audiovisual information already present in video to\n",
      "improve self-supervised representation learning? To answer this question, we\n",
      "study various pretraining architectures and objectives within the masked\n",
      "autoencoding framework, motivated by the success of similar methods in natural\n",
      "language and image understanding. We show that we can achieve significant\n",
      "improvements on audiovisual downstream classification tasks, surpassing the\n",
      "state-of-the-art on VGGSound and AudioSet. Furthermore, we can leverage our\n",
      "audiovisual pretraining scheme for multiple unimodal downstream tasks using a\n",
      "single audiovisual pretrained model. We additionally demonstrate the\n",
      "transferability of our representations, achieving state-of-the-art audiovisual\n",
      "results on Epic Kitchens without pretraining specifically for this dataset.\n",
      "\n",
      "---\n",
      "# CMAE-V: Contrastive Masked Autoencoders for Video Action Recognition\n",
      "\n",
      "## Authors\n",
      "Cheng-Ze Lu, Xiaojie Jin, Zhicheng Huang, Qibin Hou, Ming-Ming Cheng, Jiashi Feng\n",
      "\n",
      "## Categories\n",
      "cs.CV\n",
      "\n",
      "## Publication Details\n",
      "- Published: January 15, 2023\n",
      "- arXiv ID: 2301.06018v1\n",
      "\n",
      "\n",
      "\n",
      "## Abstract\n",
      "Contrastive Masked Autoencoder (CMAE), as a new self-supervised framework,\n",
      "has shown its potential of learning expressive feature representations in\n",
      "visual image recognition. This work shows that CMAE also trivially generalizes\n",
      "well on video action recognition without modifying the architecture and the\n",
      "loss criterion. By directly replacing the original pixel shift with the\n",
      "temporal shift, our CMAE for visual action recognition, CMAE-V for short, can\n",
      "generate stronger feature representations than its counterpart based on pure\n",
      "masked autoencoders. Notably, CMAE-V, with a hybrid architecture, can achieve\n",
      "82.2% and 71.6% top-1 accuracy on the Kinetics-400 and Something-something V2\n",
      "datasets, respectively. We hope this report could provide some informative\n",
      "inspiration for future works.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "def get_paper_summary(arxiv_url) -> str:\n",
    "    \"\"\"\n",
    "    Get the summary of a paper from arxiv-txt.org\n",
    "    \"\"\"\n",
    "    assert arxiv_url.startswith(\"https://arxiv.org/\"), f\"Invalid arxiv url: {arxiv_url}, must start with https://arxiv.org/\"\n",
    "    arxiv_txt_url = arxiv_url.replace(\"arxiv.org/\", \"arxiv-txt.org/raw/\")\n",
    "    response = requests.get(arxiv_txt_url)\n",
    "    return response.text\n",
    "\n",
    "\n",
    "def get_summaries(paper_list) -> str:\n",
    "    \"\"\"\n",
    "    Get the summaries of a list of papers\n",
    "    \"\"\"\n",
    "    summary_list = []\n",
    "    for paper in paper_list:\n",
    "        print(f\"Getting summary for {paper}:\")\n",
    "        summary = get_paper_summary(paper)\n",
    "        summary_list.append(summary)\n",
    "    return \"\\n---\\n\".join(summary_list)\n",
    "\n",
    "paper_summaries = get_summaries(paper_list)\n",
    "print(\"Summaries:\\n---\\n\")\n",
    "print(paper_summaries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get the lit review from the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install litellm for LLM calls\n",
    "!pip -q install litellm\n",
    "\n",
    "import os\n",
    "import litellm\n",
    "\n",
    "def get_completion(messages, model: str = \"gpt-4o-mini\") -> str:\n",
    "\n",
    "    response = litellm.completion(model=model, messages=messages)\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "model = \"gpt-4o-mini\"  # Replace with any litellm supported model, make sure to set OPENAI_API_KEY\n",
    "\n",
    "system_prompt = f\"\"\"\n",
    "You are a helpful assistant that reviews papers.\n",
    "You are given a list of papers and their abstracts.\n",
    "Your goal is to review the papers for a research paper on the given topic.\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "user_prompt = f\"\"\"This will be a paragraph in a scientific paper.\n",
    "Explain what Masked Autoencoders are, why they are important, and the different innovations listed in the follow-up papers.\n",
    "Cite the papers and their different contributions to the field of Masked Autoencoders.\n",
    "\n",
    "Here are the relevant papers and their abstracts:\n",
    "{paper_summaries}\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "messages = [\n",
    "    {\"role\": \"system\", \"content\": system_prompt},\n",
    "    {\"role\": \"user\", \"content\": user_prompt},\n",
    "]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Masked Autoencoders (MAEs) are a class of self-supervised learning models that have gained prominence for their ability to learn robust representations by reconstructing masked portions of input data. Initially popularized in the context of images, these models have been extended to spatiotemporal domains, particularly for video data. The significance of MAEs lies in their capacity to efficiently utilize unlabeled data, enabling the extraction of meaningful patterns without costly annotation efforts. These approaches have shown remarkable performance in various computer vision tasks, establishing a framework for learning that parallels successful techniques in natural language processing, such as BERT.\n",
      "\n",
      "Recent innovations in the field of MAEs have built upon this foundational architecture to address specific limitations and enhance representation learning. For instance, **Feichtenhofer et al. (2022)** introduced a spatiotemporal extension of MAEs, demonstrating that high masking ratios (up to 90%) lead to significant improvements in representation quality and computational efficiency for video datasets, outperforming traditional supervised methods (Feichtenhofer et al., 2022). Subsequently, **Wu et al. (2023)** proposed DropMAE, which incorporates spatial-attention dropout to improve temporal matching capabilities, thereby setting new state-of-the-art results in visual object tracking and segmentation tasks (Wu et al., 2023). \n",
      "\n",
      "**Bandara et al. (2022)** introduced AdaMAE, which presented an end-to-end adaptive masking strategy that samples tokens based on their semantic relevance, achieving superior performance while efficiently managing computational resources (Bandara et al., 2022). Meanwhile, **Georgescu et al. (2022)** explored the integration of audiovisual information with MAE architectures, showing enhanced performance on audiovisual classification tasks, thus exemplifying the transferability of learned representations across modalities (Georgescu et al., 2022). Lastly, **Lu et al. (2023)** developed the Contrastive Masked Autoencoder for Video Action Recognition (CMAE-V), which enhances representation strength through a hybrid architecture, achieving state-of-the-art results on benchmark action recognition datasets (Lu et al., 2023). \n",
      "\n",
      "These advancements illuminate the versatility and robustness of MAE frameworks, pushing the boundaries of self-supervised learning and establishing new benchmarks in representation learning for video and audiovisual data.\n"
     ]
    }
   ],
   "source": [
    "lit_review = get_completion(messages)\n",
    "print(lit_review)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
